Machine learning for identification of candidate video insertion object types

ABSTRACT

The method disclosure provides methods, systems and computer programs for identification of candidate video insertion object types using machine learning. Machine learning is used for at least part of the processing of the image contents of a plurality of frames of a scene of a source video. The processing includes identification of a candidate insertion zone for the insertion of an object into the image content of at least some of the plurality of frames and determination of an insertion zone descriptor for the identified candidate insertion zone, the insertion zone descriptor comprising a candidate object type indicative of a type of object that is suitable for insertion into the candidate insertion zone.

NOTICE OF COPYRIGHTS AND TRADE DRESS

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. This patent document may showand/or describe matter which is or may become trade dress of the owner.The copyright and trade dress owner has no objection to the facsimilereproduction by anyone of the patent disclosure as it appears in thePatent and Trademark Office patent files or records, but otherwisereserves all copyright and trade dress rights whatsoever.

RELATED APPLICATION INFORMATION

This application is based on and claims the benefit of priority from UKPatent Application No. GB 1714000.5, filed on Aug. 31, 2017, the contentof which is incorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to a system, method, software andapparatus for processing the image contents of a plurality of frames ofa scene of a source video and for training the system and apparatusthereof.

BACKGROUND

With the advent of digital file processing, it is possible to digitallyinsert objects (also referred to herein as “embed”) into a video.Digitally inserting objects in a video may have many benefits, forexample enhancing the visual effects of the video, or improving therealism of a video, or allowing more flexibility for the video after itis shot, meaning that fewer decisions need to be made regarding objectsto include in a scene at the stage of filming the scene. Consequently,digital object insertion is becoming increasingly common and utilised byvideo makers for all manner of purposes.

Currently, digital object insertion typically requires a number ofprocessing stages. Although described further below, these can broadlybe broken down into:

1. the detection of cuts;

2. the fusion and grouping of similar shots;

3. the detection of insertion opportunities (referred to interchangeablythroughout as insertion zones);

4. the contextual characterisation of insertion zones; and

5. the matching between insertion zones and objects for insertion.

Detection of Cuts

A programme may typically be a half hour or hour-long show, andprogramme material is decomposed into shots. Shots are a consecutivesequence of frames which do not comprise any edit points, i.e., theyusually maintain a coherence which indicates that they were recorded bya single camera.

They are delineated by cuts, where the camera usually stops recording,or the material is edited to give this impression. Broadly speaking,there are two types of cuts: “hard” cuts and “soft” cuts. A hard cut isdetected when the visual similarity between consecutive frames abruptlybreaks down, indicating an edit point or a change in camera angle, forexample. A soft cut corresponds to the beginning or end of a softtransition, for example a wipe or a fade transition, characterised by asignificant but gradual change in the visual appearance of the videoacross several frames.

First, it may be necessary to analyse the source video material (forexample, the programme material), and locate suitable scenes for objectinsertion. This is usually referred to as a pre-analysis pass, and isbest done by dividing the source video into scenes, and particularlyinto scenes shot from the same camera position. Segmentation of videomaterial into scenes may typically be performed automatically, usingshot change detection. A video analysis module may automatically detecthard and soft cuts between different shots, which correspond to hard andsoft transitions respectively.

Fusion and Grouping of Similar Shots

Once a shot or shots have been detected, continuity detection may alsobe applied in a further processing step to identify similar shots thathave been detected in the source video. In this way, when an insertionopportunity is identified in one shot, a shot similarity algorithm canidentify further shots in which the same opportunity is likely to bepresent.

Detection of Insertion Zones

Image regions in the source video content that are suitable forinsertion of additional material are referred to as insertion zones, andthese can broadly be categorised into surfaces and objects. In general,a surface may be suitable for the insertion of material. In the case ofa wall, for example, a poster might be added. In the case of a table, anobject such as a drink may be inserted. When an object is identified asan insertion zone, the opportunity for insertion material may relate torebranding any brand insignia identified on the product, replacement ofthe object with another object belonging to the same class of objects,or the addition of a further similar object in close proximity with theobject.

Detecting insertion zones can be pursued and refined through thetracking of coherently moving pixels throughout the source videomaterial. Image-based tracking techniques include but are not limited toplanar tracking algorithms to compute and model 2D transformations ofeach image in the source video.

Contextual Characterization of Insertion Zones

An operator may be required to assess the identified insertion zone andprovide context for the possible additional material which may beinserted therein. With the rapid rise in the amount of digital videocontent which is being broadcast or streamed via the internet, the factthat a human operator is not able to process insertion opportunities toidentify context much faster than in real time may be a problem.

Matching Between Insertion Zones and Product Categories

It is not enough to merely identify the insertion opportunities throughpattern recognition processes, there may also need to be someintelligence applied when selecting the material which is to be insertedinto the video content.

For an instance of object insertion not to detract from the viewingexperience, it should make sense within the context of the source videocontent into which it is placed. If a scene takes place in a kitchen,for example, additional content to be placed in that scene should berelevant to the objects that the viewer would expect to see in thatlocation. For example, one would perhaps not expect to see a perfumebottle located on a kitchen side board next to a kettle. Much moresuitable in the context described might be a jar of coffee. Likewise abathroom scene is suitable for the placement of bathroom or hygienerelated items, rather than groceries. Consequently, an operator may berequired to assess the scene to select a particular object or categoryof objects that would be suitable for insertion in any identifiedinsertion zone. Again, the fact that a human operator is not able toprocess insertion opportunities to identify context much faster than inreal time may be a problem.

It may be appreciated from the above that the identification ofinsertion zone opportunities and suitable objects for insertion maytypically be a time consuming, multi-stage process that may limit thevolume of video material that can be analysed.

SUMMARY

In a first aspect of the present disclosure, there is provided a systemcomprising: a candidate insertion zone module configured to: receive aplurality of frames of a scene of a source video; and process, at leastin part using machine learning, image contents of the plurality offrames to: identify a candidate insertion zone for the insertion of anobject into the image content of at least some of the plurality offrames; and determine an insertion zone descriptor for the identifiedcandidate insertion zone, the insertion zone descriptor comprising acandidate object type indicative of a type of object that is suitablefor insertion into the candidate insertion zone.

The candidate insertion zone module may comprise: an identificationsub-module configured to perform the identification of the candidateinsertion zone and the determination of the insertion zone descriptorfor the identified candidate insertion zone, and to: determine, for atleast some of the pixels of the plurality of frames of the scene, aninsertion probability vector comprising a probability value for each ofa plurality of insertion labels, each probability value being indicativeof the likelihood that the type of insertion indicated by thecorresponding insertion label is applicable to the pixel.

The plurality of insertion labels may comprise a label indicative of thepixel not being suitable for insertion of an object; and one or morelabels indicative of a corresponding one or more types of object.

The candidate insertion zone may comprise a plurality of pixels havinginsertion probability vectors that all have a maximum argument ofprobability values corresponding to a label that is indicative of thecandidate object type.

The candidate insertion zone module may comprise: a scene descriptorsub-module configured to process, using machine learning, image contentsof at least some of the plurality of frames to determine a scenedescriptor, wherein determination of the candidate object type is basedat least in part on the scene descriptor.

Identification of the candidate insertion zone may be based at least inpart on the scene descriptor.

The scene descriptor may comprise at least one global descriptor,wherein each global context descriptor is indicative of any one of:Scene locale; Mood; Demographic; Human Action; Time of day; Season ofthe year; Weather; and/or Filming Location.

The scene descriptor sub-module may be further configured to: receiveaudio content relating to the scene of the source video; and determinethe scene descriptor based at least in part on the received audiocontent.

The scene descriptor may comprise at least one regional contextdescriptor indicative of an identified entity in the scene. The at leastone regional context descriptor may be indicative of an identifiedentity in the scene being any one of: a human; an animal; a surface; oran object.

The scene descriptor sub-module may be configured to process, usingmachine learning, image contents of the plurality of frames todetermine, for at least some of the pixels of the plurality of frames ofthe scene, a regional context probability vector comprising aprobability value for each of a plurality of regional context labels,each probability value being indicative of the likelihood that the typeof entity indicated by the corresponding regional context label isapplicable to the pixel.

The plurality of regional context labels may comprise: a labelindicative of the pixel not relating to anything; and at least one of:one or more labels indicative of a human; one or more labels indicativeof an animal; one or more labels indicative of an object; and/or one ormore labels indicative of a surface.

The candidate insertion zone module may further comprise: a databasecomprising a contextually indexed library of types of insertion object;wherein determining the candidate object type is based at least in parton the library of types of insertion object and the scene descriptor.

In an alternative, the candidate insertion zone module may furthercomprise: an insertion zone and insertion object identificationsub-module configured to identify the candidate insertion zone and thecandidate object types by processing, using machine learning, imagecontents of the plurality of frames to determine, for at least some ofthe pixels of the plurality of frames of the scene, an insertionprobability vector comprising a probability value for each of aplurality of insertion labels, each probability value being indicativeof the likelihood that the type of insertion corresponding insertionlabel is applicable to the pixel. The plurality of insertion labels maycomprise: a label indicative of the pixel not being suitable forinsertion of an object; and one or more labels indicative of acorresponding one or more types of object being suitable for insertionin the pixel. The candidate insertion zone may comprise a plurality ofpixels having insertion probability vectors that all have a maximumargument of probability values corresponding to a label that isindicative of the candidate object type.

In any of the system implementations identified above, the candidateinsertion zone module may further comprise a post-processing sub-moduleconfigured to determine a time duration of the candidate insertion zoneacross the plurality of frames and/or a size of the candidate insertionzone.

The insertion zone descriptor may further comprise at least one of thetime duration of the candidate insertion zone across the plurality offrames and/or the size of the candidate insertion zone.

The post-processing sub-module may be further configured to determine aVideo Impact Score based at least in part on the time duration of thecandidate insertion zone across the plurality of frames and/or a size ofthe candidate insertion zone.

In any of the system implementations identified above, the system mayfurther comprise: a segmentation module configured to: generate aninsertion zone suggestion frame comprising a frame of the plurality offrames overlaid with a visualisation of the candidate insertion zone.

In any of the system implementations identified above, the system mayfurther comprise: an object insertion module configured to: select anobject for insertion based on the candidate object type; and generate anobject insertion suggestion frame comprising a frame of the plurality offrames and the selected object inserted in the candidate insertion zone.

In any of the system implementations identified above, the candidateinsertion zone module may be further configured to: receive feedbackfrom an operator, wherein the feedback is indicative of the suitabilityof the identified candidate insertion zone and/or the candidate objecttype for the image contents of the plurality of frames; and modify themachine learning based at least in part on the feedback.

The system may further comprise a final insertion module configured toreceive an object or additional material for insertion into the scene ofthe source video and generate output material comprising at least partof the source video and the received object or additional materialinserted into the candidate insertion zone, wherein the received objector additional material is of the type indicated by the candidate objecttype.

In a second aspect of the present disclosure, there is provided a methodof processing the image contents of a plurality of frames of a scene ofa source video, the method comprising: receiving the plurality of framesof the scene of the source video; and processing, at least in part usingmachine learning, image contents of the plurality of frames to: identifya candidate insertion zone for the insertion of an object into the imagecontent of at least some of the plurality of frames; and determine aninsertion zone descriptor for the identified candidate insertion zone,the insertion zone descriptor comprising a candidate object typeindicative of a type of object that is suitable for insertion into thecandidate insertion zone.

In a third aspect of the present disclosure, there is provided acomputer program for carrying out the method of the second aspect whenexecuted on the processor of an electronic device.

In a fourth aspect of the present disclosure, there is provided anelectronic device comprising: a memory for storing the computer programof the third aspect; and a processor for executing the computer programof the third aspect.

In a fifth aspect of the present disclosure, there is provided a methodof training a candidate insertion zone module to identify candidateinsertion zones and one or more candidate objects for insertion in ascene of a source video, the method comprising: receiving a trainingcorpus comprising a plurality of images, each annotated withidentification of at least one insertion zone and one or more candidateobject types for each insertion zone; and training the candidateinsertion zone module using machine learning and the training corpus toprocess image contents of a plurality of frames of the source video to:identify a candidate insertion zone for the insertion of an object intothe image content of at least some of the plurality of frames; anddetermine an insertion zone descriptor for the identified candidateinsertion zone, the insertion zone descriptor comprising one or morecandidate object types indicative of one or more types of object thatare suitable for insertion into the candidate insertion zone.

At least some of the plurality of images in the training corpus may befurther annotated with a scene descriptor, and wherein the candidateinsertion zone module may be further trained using machine learning to:identify at least one scene descriptor for the image content of at leastsome of the plurality of frames; and determine the one or more candidateobject types based at least in part on the identified at least one scenedescriptor.

The method of the fifth aspect may further comprise determining one ormore scene descriptors for at least some of plurality of images in thetraining corpus using a trained machine learning module configured toidentify a scene descriptor by processing the content of an image;wherein training the candidate insertion zone module using machinelearning further comprises training the candidate insertion zone moduleto: identify at least one scene descriptor for the image content of atleast some of the plurality of frames; and determine the one or morecandidate object types based at least in part on the identified at leastone scene descriptor.

Aspects of the Disclosure

Non-limiting aspects of the disclosure are set out in the followingnumbered clauses.

-   1. A system comprising:

a candidate insertion zone module configured to:

-   -   receive a plurality of frames of a scene of a source video; and    -   process, at least in part using machine learning, image contents        of the plurality of frames to:    -   identify a candidate insertion zone for the insertion of an        object into the image content of at least some of the plurality        of frames; and    -   determine an insertion zone descriptor for the identified        candidate insertion zone, the insertion zone descriptor        comprising one or more candidate object types indicative of one        or more types of object that are recommended for insertion into        the candidate insertion zone.

-   2. The system of clause 1, wherein the candidate insertion zone    module further comprises:

an insertion zone and insertion object identification sub-moduleconfigured to identify the candidate insertion zone and the candidateobject types by processing, using machine learning, image contents ofthe plurality of frames to determine, for each of at least some of thepixels of the plurality of frames of the scene, an insertion probabilityvector comprising a probability value for each of a plurality ofinsertion labels, each probability value being indicative of theprobability that the corresponding insertion label is applicable to thepixel.

-   3. The system of clause 2, wherein the plurality of insertion labels    comprises:

a label indicative of the pixel not being suitable for insertion of anobject; and

one or more labels indicative of a corresponding one or more types ofobject being suitable for insertion in the pixel.

-   4. The system of clause 2 or clause 3, wherein the candidate    insertion zone comprises a plurality of pixels having insertion    probability vectors that all have a maximum argument of probability    values corresponding to a label that is indicative of the one or    more candidate object types.-   5. The system of clause 1, wherein the candidate insertion zone    module further comprises:

a scene descriptor sub-module configured to process, using machinelearning, image contents of at least some of the plurality of frames todetermine a scene descriptor;

a database comprising a contextually indexed library of types ofinsertion object; and

an identification sub-module configured to:

-   -   receive the scene descriptor from the scene descriptor        sub-module;    -   identify, using machine learning, the candidate insertion zone        using the scene descriptor; and    -   determine, using machine learning, the candidate insertion        object using at least the library of types of insertion object        and the scene descriptor.

-   6. The system of clause 5, wherein the machine learning sub-module    is further configured to:

receive audio content relating to the scene of the source video; and

determine the scene descriptor based at least in part on the receivedaudio content.

-   7. The system of clause 5 or clause 6, wherein the scene descriptor    comprises at least one global descriptor, wherein each global    context descriptor is indicative of any one of:

Scene locale;

Mood;

Demographic;

Human Action;

Time of day;

Season of the year.

-   8. The system of any of clauses 5 to 7, wherein the scene descriptor    comprises at least one regional context descriptor indicative of an    identified entity in the scene.-   9. The system of any of clauses 5 to 8, wherein the identification    sub-module is configured to determine based on the scene descriptor    and library of types of insertion object, for each of at least some    of the pixels of the plurality of frames of the scene, an insertion    probability vector comprising a probability value for each of a    plurality of insertion labels, each probability value being    indicative of the probability that the corresponding insertion label    is applicable to the pixel.-   10. The system of clause 9, wherein the plurality of insertion    labels comprises:

a label indicative of the pixel not being suitable for insertion of anobject; and

one or more labels indicative of a corresponding one or more types ofobject being suitable for insertion in the pixel.

-   11. The system of any preceding clause, wherein the candidate    insertion zone module further comprises a post-processing sub-module    configured to determine a time duration of the candidate insertion    zone across the plurality of frames and/or a size of the candidate    insertion zone.-   12. The system of clause 11, wherein the insertion zone descriptor    further comprises at least one of the time duration of the candidate    insertion zone across the plurality of frames and/or the size of the    candidate insertion zone.-   13. The system of clause 11 or clause 12, wherein the    post-processing sub-module is further configured to determine a    Video Impact Score based at least in part on the time duration of    the candidate insertion zone across the plurality of frames and/or a    size of the candidate insertion zone.-   14. The system of any preceding clause, further comprising:

a segmentation module configured to:

generate an insertion zone suggestion frame comprising a frame of theplurality of frames overlaid with a visualisation of the candidateinsertion zone and at least one of the one or more candidate objecttypes.

-   15. The system of any preceding clause, further comprising:

an object insertion module configured to:

select an object for insertion based on the one or more candidate objecttypes; and

generate an object insertion suggestion frame comprising a frame of theplurality of frames and the selected object inserted in the candidateinsertion zone.

-   16. A method of processing the image contents of a plurality of    frames of a scene of a source video, the method comprising:

receiving the plurality of frames of the scene of the source video; and

processing, at least in part using machine learning, image contents ofthe plurality of frames to:

-   -   identify a candidate insertion zone for the insertion of an        object into the image content of at least some of the plurality        of frames; and    -   determine an insertion zone descriptor for the identified        candidate

insertion zone, the insertion zone descriptor comprising one or morecandidate object types indicative of one or more types of object thatare recommended for insertion into the candidate insertion zone.

-   17. A computer program for carrying out the method of clause 16 when    executed on the processor of an electronic device.-   18. An electronic device comprising:

a memory for storing the computer program of clause 17; and

a processor for executing the computer program of clause 17.

-   19. A method of training a candidate insertion zone module to    identify candidate insertion zones and one or more candidate objects    for insertion in a scene of a course video, the method comprising:

receiving a training corpus comprising a plurality of images, eachannotated with identification of at least one insertion zone and one ormore candidate object types for each insertion zone; and

training the candidate insertion zone module using machine learning andthe training corpus to process image contents of a plurality of framesof the source video to:

-   -   identify a candidate insertion zone for the insertion of an        object into the image content of at least some of the plurality        of frames; and    -   determine an insertion zone descriptor for the identified        candidate insertion zone, the insertion zone descriptor        comprising one or more candidate object types indicative of one        or more types of object that are recommended for insertion into        the candidate insertion zone.

-   20. The method of clause 19, wherein at least some of the plurality    of images in the training corpus are each further annotated with a    scene descriptor, and wherein the candidate insertion zone module is    further trained using machine learning to:

identify at least one scene descriptor for the image content of at leastsome of the plurality of frames; and

determine the one or more candidate object types based at least in parton the identified at least one scene descriptor.

-   21. The method of clause 19, further comprising:

determining one or more scene descriptors for at least some of pluralityof images in the training corpus using a trained machine learning moduleconfigured to identify a scene descriptor by processing the content ofan image; wherein

training the candidate insertion zone module using machine learningfurther comprises training the candidate insertion zone module to:

-   -   identify at least one scene descriptor for the image content of        at least some of the plurality of frames; and    -   determine the one or more candidate object types based at least        in part on the identified at least one scene descriptor.

DRAWINGS

Further features and advantages of the present disclosure will becomeapparent from the following description of an embodiment thereof,presented by way of example only, and by reference to the drawings,wherein like reference numerals refer to like parts, and wherein:

FIG. 1 shows an example schematic representation of a system inaccordance with an aspect of the present disclosure;

FIG. 2 shows an example process executed by the system of FIG. 1;

FIG. 3 shows an example list of types of object for insertion into acandidate insertion zone;

FIG. 4 shows an example insertion zone suggestion frame;

FIG. 5 shows an example objection insertion suggestion frame;

FIG. 6 shows a first example schematic representation of a configurationof the candidate insertion zone module of the system of FIG. 1;

FIG. 7a shows example attributes that may be used to describe a detected“Human” more accurately;

FIG. 7b shows example attributes that may be used to describe a detected“Object” more accurately;

FIG. 7c shows example attributes that may be used to describe a detected“Surface” more accurately;

FIG. 7d shows example attributes that may be used to describe the“Locale” of a scene;

FIG. 8 shows example steps of a process for the training the machinelearning of the scene descriptor sub-module of the system of FIG. 1;

FIG. 9 shows a second example schematic representation of aconfiguration of the candidate insertion zone module of the system ofFIG. 1;

FIG. 10 shows an example representation of a training system fortraining the candidate insertion zone module of the system of FIG. 1;and

FIG. 11 shows the intermediate results of a CNN at different stages uponfeeding an image.

DETAILED DESCRIPTION

The present disclosure relates to a technique for using machine learningto identify insertion zones in a video scene and corresponding candidatetypes of object for insertion into the insertion zone. Candidate typesof object are types of object that are suitable for insertion, and maybe, for example, classes of object such as “soda bottle”, “alcoholbottle”, “vehicle”, “cell phone”, etc., or may be more specific, such asparticular brands for particular objects.

Large-Scale Generation of Inventory of Insertion Opportunities

By using machine learning to process the image contents of a pluralityof frames to identify a candidate insertion zone and a correspondinginsertion zone descriptor comprising one or more candidate object types,the speed of identification of insertion zone opportunities and suitableobjects for insertion may be significantly increased. In particular, anoperator may straightforwardly review the candidate insertion zone andrecommended types of object for insertion, without having to make anyanalysis of the contents of the scene themselves. The one or moreinsertion zone descriptors can very quickly give an indication of whatsort of objects may be inserted into a scene (and optionally for howlong they may be visible), at which point further investigation and/orobject insertion may take place. For example, a source video maycomprise eight different scenes, and one or more candidate insertionzones and corresponding descriptors may be returned for each. Thus,without any operator time or effort, it can very quickly be understoodwhich scenes may be suitable for object insertion and what type ofobjects could be inserted into those scenes. Further processing and/oroperator time may then be focussed only on those scenes that have themost promise (for example, where inserted objects will be visible forthe longest and/or that are suitable for object types that are ofparticular interest, such as types of object that a director hasindicated they would like to see inserted into the source video, etc).Consequently, the increasingly large volume of video content that isbeing generated may be assessed more rapidly and operator time focussedonly on the most suitable scenes for object insertion.

Workflow

FIG. 1 shows an example schematic representation of a system 100 inaccordance with an aspect of the present disclosure. The system 100comprises a candidate insertion zone module 110, a scene detectionmodule 120, a segmentation module 130, an objection insertion module 140and a database 150.

FIG. 2 shows an example process executed by the system 100 foridentifying at least one insertion zone for a scene of the source videoand determining a corresponding insertion zone descriptor.

In step S210, the scene detection module 120 obtains a source video. Thesource video may comprise one or more digital files and the scenedetection module 120 may obtain the source video, for example, via ahigh speed computer network connection, the internet, or from computerreadable hardware storage device. The source video comprises frames ofvideo material, which may be grouped together into “shots” or “scenes”if recorded by the same camera or set in a particular location.

The scene detection module 120 may perform pre-analysis on the sourcevideo to create a sequence of similar shots or scenes which may besuitable for object insertion. The pre-analysis may be fully automatedin that it does not involve any human intervention. Pre-analysis maycomprise using a shot detection function to identify the boundariesbetween different shots in the source video. For example, the scenedetection module 120 may automatically detect “hard” and “soft” cutsbetween different shots, which correspond to hard and soft transitionsrespectively. Hard cuts correspond to an abrupt change in visualsimilarity between two consecutive frames in the source video. Soft cutscorrespond to the beginning or the end of a soft transition (for examplewipe and cross fading transitions), which may be characterised by asignificant but gradual change in visual appearance across severalframes. Further pre-analysis techniques known in the art may beemployed, such as continuity detection, point tracking or planartracking, 3D tracking, autokeying, region segmentation etc.

In Step S220, the candidate insertion zone module 110 processes thecontents of a plurality of frames of a scene identified by the scenedetection module 120. It will be appreciated at this point that whilstthe system 100 represented in FIG. 1 comprises the scene detectionmodule 120, the scene detection module 120 is optional and in analternative implementation the candidate insertion zone module 110 mayreceive a plurality of frames of a scene of the source video from anentity outside of the system 100, for example via a high speed computernetwork connection, the internet, or from computer readable hardwarestorage device.

The candidate insertion zone module 110 processes the contents of aplurality of frames of a scene of the source video to identify one ormore candidate insertion zones in the image content of the frames. Thecontent of all of the plurality of frames of a scene may be processed,or a subset of the plurality of frames (for example, processing speedsmay be increased by analysing fewer than all of the plurality of frames,such as processing every second frame, or analysing the similarity ofthe frames to identify groups of similar frames within a scene andidentifying only one, or some but not all, of the frames in each similargroup, etc.). Each candidate insertion zone is suitable for theinsertion of an object (or objects) into the image content of at leastsome of the plurality of frames of the scene. The candidate insertionzone module 110 also determines an insertion zone descriptor for each ofthe identified candidate insertion zones. Each insertion zone descriptorcomprises one or more candidate object types indicative of one or moretypes of object that are suitable for insertion into the correspondingcandidate insertion zone (for example, the candidate object types may beindicative of a recommendation, or suggestion, or prediction of one ormore types of object for insertion into the corresponding candidateinsertion zone). The insertion zone descriptor may also comprise furtherinformation indicative of the duration of the candidate insertion zone(for example, one or more of: the amount of time for which the candidateinsertion zone is present during the scene, the size of the insertionzone, the centralness with respect to the image, etc.). Further detailsof different ways in which the candidate insertion zone module 110 maybe configured to determine the insertion zone descriptor are explainedlater.

A candidate insertion zone is a region of the image content of a scenethat is suitable for the insertion of an object. As explained earlier, acandidate insertion zone may correspond to a table in the image contentof the scene, which may be suitable for the insertion of any type ofobject that might be placed on a table, for example a lamp or a bottleof soda. In an alternative, a candidate insertion zone may correspond toa wall, which may be suitable for the insertion of a poster. In analternative, a candidate insertion zone may correspond to an object inthe scene, for example a jar of coffee or a vehicle, which may besuitable for the insertion of a branding alteration object, in order tochange the branding of the object in the scene.

As explained above, the insertion zone descriptor may compriseinformation indicative of the duration of the insertion zone. Theduration of a candidate insertion zone is the time for which thecandidate insertion zone is present within the scene. By way ofnon-limiting example, during a scene which lasts for 30 seconds, acharacter may open the door of a refrigerator, revealing a shelf withinthe refrigerator which may be identified as a candidate insertion zone.Five seconds later, the character may close the door of therefrigerator. In this particular example, the duration of the candidateinsertion zone is five seconds, since it is visible within the scene foronly five seconds. Information in the insertion zone descriptor may beindicative of the duration of the candidate insertion zone in anysuitable way, for example by indicating the time for which the insertionzone is present within the scene in units of hours, and/or minutes,and/or seconds, and/or milliseconds, etc, or by indicating the number offrames of the scene in which the insertion zone is present (from whichthe time duration can be derived using the frame rate of the sourcevideo), etc.

The one or more candidate object types may take any suitable form,depending on the particular implementation of the candidate insertionzone module 110 and/or the requirements of the owner/operator of thesystem 100. For example, the one or more candidate object types maycomprise particular categories of objects that may be inserted into thecandidate insertion zone. An example list of 20 different categories ofobjects is given in FIG. 3, from which the one or more candidate objecttypes may be selected (for example, the candidate insertion zone may bea counter in a kitchen and the one or more candidate object types maycomprise “Food; Soft Drinks; Hot Beverages”. The one or more candidateobject types may additionally or alternatively be indicative ofparticular candidate objects for insertion, for example “Brand X SodaCan; Brand Y Coffee Bag; Brand Z Kettle; etc.).

Having identified the one or more candidate insertion zones andcorresponding one or more insertion zone descriptors, in Step S230 thecandidate insertion zone module 110 may output an identification of thecandidate insertion zones and the one or more insertion zone descriptorsfrom the system 100. Additionally, or alternatively, in Step S230, thecandidate insertion zone module 110 may pass the identification of theone or more candidate insertion zones and insertion zone descriptors tothe segmentation module 130 and/or objection insertion module 140.

In optional step S240, the segmentation module 130 selects a frame fromthe scene that includes the candidate insertion zone (for example, itmay select any arbitrary frame that includes the candidate insertionzone, or the first frame in the scene that includes the candidateinsertion zone, or the middle frame in the scene that includes thecandidate insertion zone, or the last frame in the scene that includesthe candidate insertion zone, etc.) and overlay a visualisation of thecandidate insertion zone on the selected frame in order to create aninsertion zone suggestion frame. Overlaying the visualisation of thecandidate insertion zone may be performed, for example, based on pixellabelling, wherein the candidate insertion zone module 110 has labelledthe pixels in the frames of the scene to identify each pixel as beingpart of a candidate insertion zone or not, such that the segmentationmodule may readily identify the boundaries of any candidate insertionzones. The insertion zone suggestion frame may also comprise avisualisation of the one or more candidate object types (for example,text overlaid on the frame identifying the one or more candidate objecttypes) and/or a visualisation of any of the other information in theinsertion zone descriptor, such as text overlaid on the frameidentifying the amount of time and/or number of frames for which thecandidate insertion zone is present in the scene). The overlayvisualisation of a candidate insertion zone may take the form of acoloured area on the image content of a frame of the scene, theboundaries of which correspond with the boundaries of the candidateinsertion zone.

FIG. 4 shows an example frame from a scene 410 and an example insertionzone suggestion frame 420, which is the same as the example frame fromthe scene 410 but with a visualisation of a candidate insertion zone 425overlaid on the scene. As will be appreciated, the insertion zonesuggestion frame 420 may help an operator to quickly understand thecharacteristics and possibilities of the candidate insertion zone 425,for example how prominent it may be within the scene, what sort ofobjects may be suitable for insertion and/or how long those objects maybe visible within the scene, etc. Consequently, the speed with which anassessment of the potential value of the candidate insertion zone andsubsequent object insertion may be considerably increased, since asource video or a scene of a source video may be input to the system 100and a readily intelligible representation of a candidate insertionzone(s) and object insertion opportunities within that candidateinsertion zone(s) quickly output from the system 100.

In optional step S250, the object insertion module 140 performs asimilar operation to the segmentation module 130, except that ratherthan generate an insertion zone suggestion frame 420, it generates anobject insertion suggestion frame. This may be much the same as theinsertion zone suggestion frame 420, but rather than overlay avisualisation of the candidate insertion zone, the object insertionsuggestion frame may comprise a frame of the scene with an objectinserted into the candidate insertion zone. In this way, a mock-up ofthe insertion opportunity may be created.

To this end, the object insertion module 140 may be configured to selectan object for insertion from the database 150, which may comprise alibrary of graphics of objects for insertion, based on the one or morecandidate object types, and insert the selected object into the frame.The library of graphics of objects may be indexed by object type so thatthe object to be inserted may be any object that matches the one or morecandidate object types in the insertion zone descriptor (for example, ifthe insertion zone descriptor identifies “Drink, Soft Drinks” as acandidate object type, any type of soft drink object in the database 150may be selected and inserted into the frame to create the objectinsertion suggestion frame). Optionally, the object insertion module 140may generate a plurality of different object insertion suggestionframes, each comprising a different object, such that the visualappearance of different objects inserted into the scene may be readilyappreciated. Further optionally, rather than inserting a fullrepresentation of the object, the object insertion module 140 may inserta shape (for example, a coloured box or cylinder, etc) thatapproximately matches the shape of a generic object that matches thecandidate object type. This may help with visualising how the scene mayappear after object insertion, without being limited to a specificobject that is within the candidate object type.

FIG. 5 shows an example frame from a scene 510 and an example objectinsertion suggestion frame 520, which is the same as the example framefrom the scene 510 but with a suggested object 525 inserted into thescene. Optionally, the object insertion suggestion frame 520—may alsocomprise a visualisation of any other information in the insertion zonedescriptor (for example, text overlaid on the frame identifying theamount of time and/or number of frames for which the candidate insertionzone is present in the scene). It will be appreciated that the objectinsertion suggestion frame 525 may help in the rapid visualisation ofhow the scene might appear with a suitable object inserted into it.Furthermore, if a determination is made to insert the object into thescene, it may help to speed up the process of insertion, since theoperator can very quickly understand how and where the object can beinserted.

Based on the candidate insertion zone(s) and candidate object types(s),and/or the object insertion suggestion frame and/or the insertion zonesuggestion frame, one or more objects corresponding to the type ofobject(s) indicated by the candidate object type may be inserted intothe scene of the source video, such that they appear within the imagecontent of the frames of the scene. For example, an operator may decidewhether or not to proceed with the insertion based on the candidateinsertion zone(s) and candidate object types(s), and/or the objectinsertion suggestion frame and/or the insertion zone suggestion frame.If they decide to proceed, an object(s) can be inserted according to anystandard techniques well understood by the skilled person. If theydecide not to proceed, nothing further may happen. Alternatively,insertion of an object(s) of the type indicated by the candidate objecttype(s) may automatically take place after the candidate insertionzone(s) and candidate object types(s) have been determined.

The candidate insertion zone module 110 uses machine learning techniquesto perform at least some of the steps necessary to process the imagecontents of the plurality of frames of a scene to identify at least onecandidate insertion zone in the scene and a corresponding at least oneinsertion zone descriptor. There are a number of different ways in whichthe candidate insertion zone module 110 may be configured to use machinelearning to this end, which are summarised below as an “indirectapproach” or a “direct approach”. Example configurations of thecandidate insertion zone module 110 in accordance with each of the“indirect approach” and the “direct approach” are described below withreference to FIGS. 6 and 9.

Indirect Approach

FIG. 6 shows an example schematic representation of a configuration ofthe candidate insertion zone module 110 for carrying out the “indirectapproach” of identifying a candidate insertion zone and determining aninsertion zone descriptor. The candidate insertion zone module 110comprises a scene descriptor sub-module 610, an identificationsub-module 620, a database 630 comprising a library of types ofinsertion object and a post processing sub-module 640. The database 630may be the same as database 150, or it may form part of database 150 (ordatabase 150 may form part of database 630), or may be completelyseparate from database 150.

Regional Context Descriptor

The scene descriptor sub-module 610 is configured to process the imagecontent of a plurality of frames of the scene using machine learning todetermine scene descriptors. The scene descriptors may comprise at leastone regional context descriptor and/or at least one global contextdescriptor.

A regional context descriptor may be indicative of what type of “thing”a part of the image content of the plurality of frames is. For example,an identified part within the image may be classified semantically intoany one of four classifications of regional context descriptor: (1)Human, (2) Animal, (3) Surface, (4) Object. Where a part of an image hasbeen identified as being part of one of the four classifications ofregional context descriptor, that part of the image may be characterisedmore precisely using attributes associated with that classification ofregional context descriptor.

FIG. 7a , for example, shows attributes that may be used to describe adetected “Human” more accurately. In this particular example, a “Human”may be described more accurately with two different types of attribute:Gender and Age. However, it will be appreciated that any number of othertypes of attribute may additionally or alternatively be used, forexample: ethnicity, hair colour, etc. Additionally, or alternatively,attributes may identify particular actors or characters so that they maybe tracked throughout shots of a sequence. For this purpose, one of alarge number of readily available facial recognition packages may beused to identify the characters and/or actors, using Fisher vectors, forexample. Fisher vectors are described in K. Simonyan, A. Vedaldi, A.Zisserman. “Deep Fisher networks for large-scale image classification”Proc. NIPS, 2013.

FIG. 7b , for example, shows attributes that may be used to describe adetected “Object” more accurately. Again, these attributes are shownonly by way of non-limiting example and any other suitable “Object”attributes may additionally or alternatively be used. Furthermore,whilst in this example an identified “Object” is described with only onetype of Object attribute, it may alternatively be described using two ormore different types of attribute, for example: the category of theobject (such as drinks can, magazine, car, etc) and the brand of theobject.

FIG. 7c , for example, shows attributes that may be used to describe adetected “Surface” more accurately. Again, these attributes are shownonly by way of non-limiting example and any other suitable “Surface”attributes may additionally or alternatively be used. Furthermore,whilst in this example an identified “Surface” is described with onlyone type of “Surface” attribute, it may alternatively be described usingtwo or more different types of “Surface” attribute.

Pixel Labelling for Determining Regional Context Descriptors

The machine learning sub-module 610 may be configured to determine theone or more regional context descriptors in any suitable way. In oneparticular example, it may be configured to annotate each of at leastsome of the pixels of a plurality of frames of the scene (or each of atleast some of the pixels in a plurality of frames of the scene, asexplained in more detail later), with a regional context probabilityvector. Whilst it may be preferable for each of at least some of thepixels to be annotated with a regional context probability vector forreasons of resolution, in an alternative implementation each regionalcontext probability vector may relate to a group of two or more pixels.For example, the pixels making up a frame may be grouped into a seriesof sub-sets, each sub-set comprising two or more pixels. In this case,each sub-set may be annotated with a regional context probabilityvector. Consequently, the machine learning sub-module 610 may beconfigured to annotate at least some of the pixels (either individuallyor in sub-set groups) with regional context probability vectors. Theregional context probability vector may comprise a probability value foreach of a plurality of regional context labels, each probability valuebeing indicative of the likelihood that the type of entity indicated bythe corresponding regional context label is applicable to that pixel(s)(for example, the values in the regional context probability vector maybe indicative of a relative ‘scoring’ of each of the labels,representing the relative likelihood of each of the labels beingapplicable to that pixel(s)). A non-limiting example of a regionalcontext probability vector for a pixel is as follows:c=[0.1, 0.05, 0, 0.05, 0, 0.05, 0.05, 0.05, 0.4, 0.15, 0.1]

Each of the items in the vector c corresponds to a regional contextlabel, wherein each regional context entity is indicative of a differenttype of entity. In this particular example, the regional context labelsare:

Not a ‘thing’, Male aged under 45, Male aged over 45, Female aged under45, Female aged over 45, Animal, Table Top, Kitchen Counter top,Vehicle, Computer, Book

Thus, each of the regional context labels for the pixel in this examplehave the following probability values:

-   Not a ‘thing’=0.1-   Male aged under 45=0.05-   Male aged over 45=0-   Female aged under 45=0.05-   Female aged over 45=0-   Animal=0.05-   Table top=0.05-   Kitchen Counter top=0.05-   Vehicle=0.4-   Computer=0.15-   Book=0.1

Thus, it can be seen that there are four labels relating to the “Human”classification (each label being an attribute relating to Humans), onelabel relating to the “Animal” classification, two labels relating tothe “Surface” classification (each label being an attribute relating toSurfaces) and three labels relating to the “Object” classification (eachlabel being an attribute relating to Objects).

The “Not a ‘thing’” label indicates the likelihood that the pixel doesnot belong to any of the other regional context labels, i.e., thepixel(s) does not relate to anything. The probability of the “Not a‘thing’” label may be set to: 1 minus the sum of all of the otherprobabilities in the regional context vector. Consequently, the sum ofall probabilities in the regional context probability vector should be1.

Therefore, in this example, the regional context label having aprobability with the greatest argument (i.e., the highest probability)is ‘Vehicle’. Thus, the regional context label considered to be mostlikely applicable to the pixel(s) is ‘Vehicle’ (i.e., that pixel(s) isthought most likely to be part of a vehicle).

Whilst each of the probabilities in the regional context probabilityvector in this particular example are between 0-1, with higher valuesindicating greater likelihood, it will be appreciated that the regionalcontext probability vector may take any other suitable form that isindicative of the likelihood that the type of entity indicated by thecorresponding regional context label is applicable to a pixel(s). Forexample, a regional context probability vector may comprise probabilityvalues between 0-20, or between 10-20, or between 0-100, etc, whereineach value is indicative of the relative likelihood that the type ofentity indicated by the corresponding regional context label isapplicable to a pixel(s). It can therefore also be seen that theprobabilities need not necessarily sum to 1, but may alternatively sumto any other suitable value.

Whilst in the above there is one particular example of a regionalcontext probability vector, it will be appreciated that the machinelearning sub-module 610 may be configured to determine regional contextprobability vectors comprising any number of probability valuescorresponding to regional context labels, for example 100s or 1000s ofprobability values corresponding to 100s or 1000s of regional contextlabels.

By determining regional context probability vectors for pixels in theframes, an understanding of what ‘things’ are in the image content of aframe, and their relative positioning, may be reached. For example, aregion of the frame where all the pixels are annotated with regionalcontext probability vectors with maximum arguments of probability valuescorresponding to ‘Animal’ is likely to have an animal. A differentregion of the frame where all the pixels have regional contextprobability vectors with maximum arguments corresponding to ‘Table top’is likely to have a table top. Because the positioning of each pixel inthe frame is known, the proximity of the animal and table top may alsobe known. Thus, it can be said that the image contents of the frameincludes a table top and an animal and their proximity to each other isperceptible.

It will be appreciated that not only can the regional contextprobability vectors be used to identify what ‘things’ are in the imagecontent of a frame and their proximity to each other, it may be used todetermine how many ‘things’ are within the image content of the frame.For example, the total number of ‘things’ of any type may be determinedand/or the total number of each different type of ‘thing’ may bedetermined (for example, number of humans, number of animal, number ofsoda cans, etc, etc). This may be useful for a number of purposes, suchas determination of a global context descriptor and/or determination ofcandidate object types (as explained in more detail later).

Furthermore, pixels that are identified by the regional contextprobability vectors as being part of a surface may be indicative of acandidate insertion zone. Likewise, pixels identified by the regionalcontext probability vectors as being part of an object may also beindicative of a candidate insertion zone (since the branding of theidentified object, for example, may be changed by object insertion).Thus, the regional context probability vectors may not only providefurther information about the ‘things’ within an image content, but mayalso be used to identify potential insertion zones and their proximityto other ‘things’ identified in the image content.

Global Context Descriptor

A global context descriptor is indicative of an overall context of theimage contents of the plurality of frames. One or more different globalcontext descriptors may be determined by the machine learningsub-module, each corresponding to a different global contextclassification. Non-limiting examples of global context classificationsare: Locale, Human Action, Demographics, Mood, Time of Day, Season ofthe Year (for example, spring, summer, autumn, winter), weather, filminglocation, etc, etc.

FIG. 7d , for example, shows a set of attributes that may be used todescribe the Locale of a scene. In this example, 41 different types oflocale are listed, although it will be appreciated that the scenedescriptor sub-module 610 may be configured to determine a localecontext descriptor for a scene from a list of any number of differentlocale attributes. Furthermore, whilst the list in FIG. 7d identifiesgeneric locale attributes, more specific locale attributes mayadditionally or alternatively be used, for example particular rooms orplaces that regularly occur within a film or television series may belocale attributes, such as a particular character's bedroom, or aparticular family's kitchen, etc.

The scene descriptor sub-module 610 may determine at least one globalcontext descriptor using machine learning in any suitable way. In oneexample, for at least one frame of a scene, the scene descriptorsub-module 610 may use machine learning to determine at least one globalcontext probability vector. Each global context probability vector for aframe may correspond to a different classification of global contextdescriptor (for example, Locale, Mood, etc) and may comprise a pluralityof probabilities, each corresponding to a different global context label(each global context label being an attribute for the particularclassification of global context descriptor). Based on the examplerepresented in FIG. 7d , a global context probability vectorcorresponding to the “Locale” classification may comprise 41 probabilityvalues corresponding to the 41 different attributes listed in FIG. 7d .The probability values in the global context probability vector areindicative of the likelihood that the different listed attributes areapplicable to the scene. Each probability value may be between 0-1, ormay take any other suitable form indicative of relative likelihood, forexample values between 0-20, or between 10-20, or between 0-100, etc.The probabilities in each global context probability vector mayoptionally sum to 1, or any other suitable value. The attribute with thelargest corresponding probability argument for each global contextprobability vector may then be considered to be the attribute that bestdescribes the global context of the scene. For example, if for a globalcontext probability vector relating to Locale the maximum probabilityargument corresponds to the attribute “Outdoor Day urban street”, theglobal context descriptor may comprise “Locale{ Outdoor Day urbanstreet}”. If a global context probability vector relating to Mood themaximum probability argument corresponds to the attribute “Happy”, theglobal context descriptor may also comprise “Mood{Happy}”, etc. Thus,the global context descriptor may comprise one or more global contextprobability vectors and/or one or more chosen attributes for each typeof global context (for example, Locale{Outdoor Day urban street},Mood{Happy}, etc).

The global context descriptors may be determined using machine learningby directly determining them from processing the image content of aplurality of frames, or by deriving them from the regional contextdescriptors. For example, it may be possible to infer suitableattributes for one or more global context descriptors based on the oneor more regional context descriptors for the image contents of a frame.By way of example, if we consider the following attributes identifiedfor the regional context classifications “Object”, “Surface” and “Human”in the image content of a frame:

-   Object {sink, bottle, cereal box}-   Surface {table, counter top, wall}-   Human {woman, widow}    it may be inferred that a suitable attribute for the global context    classification “Locale” is “kitchen”. Likewise, by way of a further    example, if regional context descriptor attributes such as road and    bench are determined, it may be inferred that a suitable attribute    for the global context classification “Locale” is “outdoors”. The    number of objects identified within the image content of the frame,    particularly the number of particular types of objects, may also be    indicative of particular global context attributes.

In addition to processing the image content of a plurality of frames inorder to determine the scene descriptors, the machine learningsub-module 610 may optionally also process audio data corresponding tothe frames. This may improve the reliability of determination. Forexample, gunshots are normally perceived as being bad, and so mayprovide strong cues to the attributes of the Human Action and/or Moodclassifications of global context descriptors. Likewise, laughter mayprovide a cue to the happiness attribute of the Mood classification ofglobal context descriptors, shouting may provide a cue to the excitementattribute of the Mood classification of global context descriptors, etc.

The scene descriptors are passed to the identification sub-module 620,which uses machine learning to identify one or more candidate insertionzones in the image contents based on the scene descriptors and determinean insertion descriptor for each. They may be passed to theidentification sub-module 620 in the form of an annotated plurality offrames that are annotated with the regional context probability vectorsand/or global context probability vectors described above, and/orannotated with the most relevant scene descriptor(s) for the scene (forexample, the chosen global context attribute for each type of globalcontext, etc).

As explained earlier, the regional context probability vectors may beindicative of parts of the image contents that could be insertion zones,for example regions that relate to a “Surface” or “Object”. Throughmachine learning, the identification sub-module 620 may be able toidentify which of these regions are most suitable to be candidateinsertion zones (for example, based on their size, positioning in theframe, positioning relative to other ‘things’ in the frame identified bythe regional context descriptors, etc.).

Demographics Context Descriptor

The identification sub-module 620 may also determine one or morecandidate object types for the insertion zone descriptor for eachcandidate insertion zone. This may be determined, for example, based atleast on the scene descriptors and a library of types of insertionobject stored in the database 630 that are contextually indexed objecttypes. Thus, the candidate object types may be determined in a way thatis most suitable for the scene based on global context properties and/orregional context properties for the scene.

By way of example, the people appearing in the scene may be useful indetermining a suitable candidate object type for the scene. This may bebecause insertion objects often relate in some way to people, such thatsome insertion objects may look natural in proximity to some types ofpeople, but look unnatural in proximity to other types of people. Forexample, the general perception may be that children are more interestedin toys and adults more interested in clothes or home appliances.Therefore, if the scene descriptor includes a regional contextdescriptor in the Human classification that identifies the attribute“child”, it may be more appropriate to recommend toys for insertion intothe image contents of the frames. Consequently, the identificationsub-module 620 may learn through machine learning that candidate objecttypes that are indexed in the library with the context of childrenshould be suitable for insertion in this scene.

To consider another example, a manufacturer of soft drinks may have arange of different brands that are marketed to different categories ofconsumer. It is generally known that diet or light drinks tend to bemarketed more heavily towards women. The identification sub-module 620may recognise through machine learning that the candidate insertion zoneand the regional context descriptors and/or global context descriptorssuggest that the insertion of a soft drink might be appropriate. Forexample, the scene descriptors include a Locale descriptor of “kitchen”,a Surface of “refrigerator shelf” and an Object of “soft drinks” nearthe candidate insertion zone in the refrigerator, in which case theidentification sub-module 620 may perform a search of the contextuallyindexed library in the database 630 and identify that the insertion of asoft drink may be appropriate (candidate object type=“soft drinks”).This may be a very useful recommendation for object insertion. However,if the scene descriptor also identifies that the scene includes a woman,the search of the contextually indexed library may more specificallyidentify a particular brand(s) of soft drinks that tend to be marketedmore heavily towards women, in which case the candidate object type maybe set to that particular brand(s). In this case, the candidate objecttype is more specific and may therefore be more helpful for subsequentanalysis and/or object insertion.

It may be appreciated, therefore, that scene descriptors may becorrelated with different types of object and machine learning may beused to learn these correlations. For example, the links between thedetected instances of regional context descriptors of Locale{bedroom},Human{child}, and Surface{floor} is likely to mean that an insertionobject type of “toys/games” would be appropriate. An insertion objecttype of “DIY furnishing accessories” or “spirits/liqueurs” is unlikelyto be appropriate.

Insertion Probability Vector

The identification sub-module 620 may annotate each pixel in a pluralityof frames of a scene with an insertion probability vector a. Theinsertion probability vector a may be very similar to the regionalcontext probability vector c described above, in that it may have aplurality of probability values, with all but one of which correspondingto an object type. The remaining one probability value may correspond toa label of “not suitable for object insertion”. Each of the probabilityvalues are indicative of the likelihood that the type of insertionindicated by the corresponding insertion label is applicable to thepixel (for example, the values in the insertion probability vector maybe indicative of a relative ‘scoring’ of each of the labels,representing the relative likelihood of each of the labels beingapplicable to that pixel).

Whilst it may be preferable for each of at least some of the pixels tobe annotated with an insertion probability vector for reasons ofresolution, in an alternative each insertion probability vector mayrelate to a group of two or more pixels. For example, the pixels makingup a frame may be grouped into a series of sub-sets, each sub-setcomprising two or more pixels. In this case, each sub-set may beannotated with an insertion probability vector. Consequently, theidentification sub-module 620 may be configured to annotate at leastsome of the pixels (either individually or in sub-set groups) withinsertion probability vectors.

The probability values in the insertion probability vector may take anysuitable form. For example, they may each be a value between 0-1, 0-10,or 20-40, or 0-200, etc, etc, with higher values indicating greaterlikelihood. The sum of the probabilities in the insertion probabilityvector a may total 1, or may total any other suitable value. If theinsertion probability vector is configured to have probability valuesthat sum to 1, the value corresponding to “not suitable for objectinsertion” may be set to 1 minus the sum of all of the other probabilityvalues. This annotation may be added to the annotated version of theplurality of frames earlier received from the scene descriptorsub-module 610 (such that the plurality of frames includes scenedescriptor and insertion descriptor annotations), or may be added to a‘fresh’ version of the frames (such that the plurality of framesincludes only insertion descriptor annotations). The annotated framestherefore indicate the candidate insertion zone within the image contentof the frames and also the corresponding one or more candidate objecttypes.

Thus, a candidate insertion zone may be identified by virtue of an areawithin the image contents of the frame that comprises a plurality ofpixels having insertion probability vectors all having a maximumargument of probability values corresponding to a label that isindicative of a particular object type. That particular object type isthe candidate object type for that candidate insertion zone.

Modelling the Visual Impact Score

The post processing sub-module 640 may receive the annotated pluralityof frames in order to identify groupings of pixels that are annotatedwith insertion probability vectors where the maximum arguments of thevectors all correspond to the same label (i.e., to the same candidateobject type). It may also determine the size, location and/or durationof the candidate insertion zone in the same way. The post processingsub-module 640 may thus output from the candidate insertion zone module120 an indication of the one or more candidate object types for theidentified insertion zone and any other insertion zone descriptorinformation it has determined (for example, the size, location and/orduration of the insertion zone).

Optionally, the post-processing module 640 may also be configured todetermine a Video Impact Score (VIS) for one or more of the identifiedcandidate insertion zones. The VIS may be included as one of theinsertion zone descriptors and may be used to assess an insertion zone'spotential impact on viewers of the video. The VIS may be a multiplier tothe quality score of an object insertion opportunity value to accountfor the highly variable nature of object embedding into video content.VIS may take any suitable form, for example a number lying on a scale,such as a number between 0 and approximately 2 (although the scale maybe of any size and granularity). In reality, VIS may not be allowed tobe less than 1 and is generally between 1 and 2.

The VIS for a candidate insertion zone may be calculated based on atleast part of the insertion zone descriptor for the insertion zone, forexample based on the duration of the candidate insertion zone and/or thesize of the candidate insertion zone.

One non-limiting example technique for determining VIS is identifiedbelow. In this example, the VIS is based on combining an Exposure Scoreand a Context Score (although any other suitable function fordetermining the VIS using any one or more insertion zone descriptoritems). These two scores are a weighted combination of a number ofparameters including Brand relevance, Duration, Hero Status, Proximity,Amplification, as defined below.

Consider the following:

Calculating  Video  Impact  ScoreBETA VIS = ES + CS ES = Exposure  ScoreCS = Context  Score Calculating  Exposure  ScoreES = W_(D)f(D) + W_(s)f(S) + W_(A)A D = Quailfying  Exposure  DurationS = Average  Exposure  Size$A = {{Amplification} = \left\{ {{\begin{matrix}{0,} & \left| {amplified} \right. \\{1,} & {amplified}\end{matrix}{f(D)}} = {{{Duration}\mspace{14mu}{valuation}\mspace{14mu}{function}{f(S)}} = {{{Size}\mspace{14mu}{valuation}\mspace{14mu}{function}W} = {Weight}}}} \right.}$

The Context Score (CS) is a weighted combination of metrics specific toembedding objects (particularly branded objects) into video content,focused on providing a valuation depending on the fit between the object(or brand) and the content.

The CS may be between 0 and approximately 2 (although the scale may beof any size and granularity).

The primary term for determining the CS may be the Brand Relevance,which is used to determine whether the brand fits the context (e.g.Vodka in a bar). If there is no Brand Relevance, then the score is 0,and the CS will be 0. When we have Brand Relevance, the Context Score is1 or above, with the rest of the terms supplying boosts in value.

The Context Score may be carried out as follows, although it will beappreciated that where CS is used to determine the VIS, CS may bedetermined in any other suitable way (for example, using only one ormore of B, H and P identified below):

${CS} = \left\{ {{\begin{matrix}{0,} & {B = 0} \\{{B + {W_{H}H} + {W_{P}P}},} & {B = 1}\end{matrix}B} = {{{Brand}\mspace{14mu}{Relevance}} = \left\{ {{\begin{matrix}{0,} & \left| {match} \right. \\{1,} & {match}\end{matrix}H} = {{{Hero}\mspace{14mu}{Status}} = \left\{ {{\begin{matrix}{0,} & \left| {match} \right. \\{1,} & {match}\end{matrix}P} = {{Proximity} = \left\{ \begin{matrix}{0,} & \left| {touching} \right. \\{1,} & {touching}\end{matrix} \right.}} \right.}} \right.}} \right.$

Thus, it will be appreciated that a VIS may be determined for acandidate insertion zone in a new video based on at least some insertionzone descriptors. The VIS for a candidate insertion zone may be a usefultechnique for ranking candidate insertion zones, or filtering poorercandidate insertion zones such that the number of candidate insertionzones for a new video that meet a particular video impact requirement(for example, that have a VIS greater than a threshold value) may bereadily identified and the potential suitability for object insertionopportunities for the new video straightforwardly appreciated.

In an alternative, the post-processing module may not be used, and theidentification sub-module 620 may simply output the annotated frames, sothat any other modules or sub-modules within the system 100 (forexample, the object insertion module 140), or external to the system100, may process the annotations to recognise the candidate insertionzones and the corresponding insertion zone descriptors.

Modelling the Indirect Approach

Before the “direct approach” is described, it is worth considering somefurther details of how the scene descriptor sub-module 610 and theidentification sub-module 620 may be implemented to carry out machinelearning and in particular how they may be trained. Preferably, in the“indirect approach”, we will use Convolutional Neural Networks (CNN) forthe recognition of scene descriptors and Support Vector Machines (SVM)for the recognition of insertion descriptors.

The Convolution Neural Network: A Bioinspired Mathematical Model

CNNs may be used for the recognition of the different scene descriptors.A CNN is a network of learning units, so-called neurons. A CNN is usedto sequentially transform the initial image contents of video frame intoan interpretable feature map that summarises the image.

The CNN is biologically inspired from the feed-forward processing of thevisual information and from the layered organisation of neurons in thevisual cortex. Like the different areas of the visual cortex, neurons ina CNN are grouped into layers, each neuron within the same layerperforming the same mathematical operation.

Typically a layer in a CNN can perform either (1) a convolutionaloperation, or (2) an activation operation, or (3) pooling operation or(4) an inner product operation. The first layers of a CNN performconvolutional operations on the image with a bank 2D of convolutionfilters. They loosely model the behaviour of retinal cells in the areaV1 of the visual cortex in the sense that They behave like Gabor filtersand subsequently forwards signals into deeper areas of the visualcortex. A convolution filter also models the fact that adjacent retinalcells have overlapping receptive fields and respond similarly to anidentical visual stimulus.

Then like the V2 area and other areas of the visual cortex, subsequentlayers of the CNN build higher-level features by combining lower-levelfeatures. However caution is required in pursuing the analogy becauseartificial neural networks do not exactly replicate the biologicalprocesses of learning visual concepts.

In more detail, the scene descriptor sub-module 610 may need to betrained (1) to determine a global scene descriptor and (2) to determineregional context descriptors via pixel labelling. In order to do this,the existing corpus of video material used for training should beannotated in a similar way. In order to explain the process of trainingin more detail, it may be helpful first to introduce some definitions.

Definitions

A CNN operates on tensors. By definition a tensor is a multidimensionalarray and is used to store and represent image data and the intermediatedata transformations of the CNN, often called feature maps.

Thus an image can be represented as a 3D tensorX∈

^(C×H×W)where C, H, W respectively denote the number of image of channels, theimage height and the image width. The RGB colour value of pixel is the3D vector.

$\quad\begin{bmatrix}{x\left\lbrack {1,i,j} \right\rbrack} \\{x\left\lbrack {2,i,j} \right\rbrack} \\{x\left\lbrack {3,i,j} \right\rbrack}\end{bmatrix}$

The output of a CNN depends on the visual recognition tasks. Let usprovide some output examples.

-   -   In the image classification task, e.g., of determining the        “Locale” global context descriptor for a given image x, the        final output of a CNN is a probability vector        y=CNN(x)        where the k-th coefficient [k] quantifies the probability that        an image corresponds to class k, say a “Kitchen” locale and the        best “Locale” descriptor for the image x is determined as

$k^{*} = {\underset{k}{argmax}{y\lbrack k\rbrack}}$

In the image segmentation task, e.g., of determining the regionalcontext descriptor, the final output of a CNN is a 3D tensor ofprobability vectors where each coefficient quantifies the probabilitythat an image pixel (i,j) corresponds to class k, say a ‘Table’ pixel.Thus, the best pixel labelling is determined as the tensor defined by

${k^{*}\left\lbrack {i,j} \right\rbrack} = {\underset{k}{argmax}{{y\left\lbrack {k,i,j} \right\rbrack}.}}$

The dimensionality of tensors does not really matter as layers canoperate on tensors of any dimension. When dealing with video data asinput, CNNs are sometimes called video networks in the computer visionliterature. In practice, it is sufficient and computationally moreefficient to just use image data and exploit temporal coherence by usinga Long-Short-Term-Memory (LSTM) networks. In particular they aredesigned to deal with an infinite sequence of data.

Besides, in practice, it is more efficient to feed images by batch to aCNN than feeding one image by one. A batch of N images can berepresented by a 4D tensorX∈

^(N×C×H×W)For video data, a batch of videos are 5D tensorsX∈

^(N×T×C×H×W).

In the sequel we restrict the description to image data and we leave thereader the exercise of generalising subsequent definitions to videodata. As presented above a CNN is composed of interconnected layers. Alayer is a differentiable function. The differentiability is a centralproperty in CNN as it is a necessary condition to back propagate thegradients during the training stage.

As another analogy to physics, a CNN can be thought an electric networkwhere tensors can be thought as input or output electric signals and alayer is an electric component that filters the incoming electricsignals from incident layers.

Definition 1 We define a Convolutional Neural Network (CNN) as adirected acyclic graph G=(V,E) where each node v E V is a layer.

Classical CNNs that are successful in image classification tasks aretypically a chain of layers. Let us define the convolutional layer whichis the most important layer used in a CNN.

Definition 2 Let k be a tensor kernel in R^(N′×C′×H′×W′). Theconvolutional layer with k is defined as the function that transforms aninput tensor x∈ R^(N′×C′×H′×W′) (e.g. an image) into a tensor x*k ∈R^(N×N′×H′×W′)

${\left( {x*h} \right)\left\lbrack {n,n^{\prime},i,j} \right\rbrack} = {\sum\limits_{c^{\prime},i^{\prime},j^{\prime}}\;{{x\left\lbrack {n,{c^{\prime} - c},{i^{\prime} - i},{j^{\prime} - j}} \right\rbrack} \times {k\left\lbrack {n^{\prime},c^{\prime},i^{\prime},j^{\prime}} \right\rbrack}}}$

In words, the tensor kernel k encodes N convolutional kernel filters(i.e., N convolutional neurons) and, as an abusive simplification,convolutional layer can be as a kind of local averaging operationapplied on all image patches of size C×H×W of each image x[n., . , . ].Each feature vector y[n, . , i, j] is a vector of dimension N′ thatdescribes the pixel x[n.,i,j] of the n-th image x[n., ., .].

In the sequel, the n-th image is also denoted by x_(n) ∈ R^(C×H×W) toalleviate the notation. An important observation is that a convolutionaloperation is equivalent as a simple matrix-matrix product operation,which is how popular deep learning packages implement it. Specifically,

1. by forming a matrix φ(x) of shape HW×C′H′W′ where each row Wi+jencodes an image patch centred at (i,j) of shape C′×H′×W′; and

by reshaping the tensor kernel k into a matrix K of size C′H′W′×N′K=[vec(k ₁); . . . ; vec(k _(N′))],then we observe that Property 1 the tensor convolution is equivalent tothe matrix-matrix product

$\underset{\underset{{HW} \times N^{\prime}}{︸}}{x_{n}*k} = {\underset{\underset{{HW} \times C^{\prime}H^{\prime}W^{\prime}}{︸}}{\phi\left( x_{n} \right)} \times \underset{\underset{C^{\prime}H^{\prime}W^{\prime} \times N^{\prime}}{︸}}{K}}$and the derivative of the convolution w.r.t. to the tensor kernel k is

$\frac{{\partial x_{n}}*k}{\partial K} = {\phi\left( x_{n} \right)}$

Thus, the tensor convolution of a batch of N images x with kernel kconsists in applying N matrix-matrix products, which efficient linearalgebra packages implement very efficiently. Note that the φ functioncan be implemented with the famous im2col function in MATLAB or Python.

At each iteration of the training stage, the gradient of the tensorconvolution is computed to update the weights of kernel k and is backpropagated to previous layers because of the chain rule.

Let us define a global scene probability vector.

Definition 3 A global scene probability vector is defined as vector ofarbitrary dimension where the k-th vector entry is the confidence valueto an attribute of just one classification of global context descriptor.

For example entries of the vector can correspond to ‘Kitchen’,‘Living-Room’, ‘Urban’ Locale descriptors and so on.

To identify the regional context descriptors at each pixel, it isassumed that we have a training set of images x_(n) where each pixelx_(n)[., i,j] is annotated with a probability vector y_(n)[., i,j]. Thisleads us to define a regional scene probability tensor.

Definition 4 A regional context probability tensor c is defined as atensor of probability vectors in [0,1]^(N×C′×H×W) where c[n,k,i,j]quantifies a confidence value for the k-th regional descriptor for eachpixel x_(n)[., i, j].

Notice that the regional context probability tensor has the same widthand height as the image tensor x. Only the depth of tensor differs.

Multi-Objective Loss Function and Weight Sharing. One dedicated CNN maybe trained to predict each type of global context descriptor (locale,mood and so on). Classically, the training stage is formulated as aparameter estimation problem. To this end, a differentiable lossfunction l(CNN(x), y) is needed to measure the error between theestimated probability vector CNN(x) and the ground-truth probabilityvector y where each entry y[k] is 0 everywhere except for the one atsome index k where the value is 1.

Then the training stage minimises the sum of errors over all the data(x_(i), y_(i)), i=1 . . . , N, in the training data:

$\min\limits_{k_{1},\ldots\mspace{14mu},k_{V}}{\sum\limits_{i = 1}^{N}\;{\ell\left( {{{CNN}\left( {{x_{i};k_{1}},\ldots\mspace{14mu},k_{V}} \right)},c_{i}} \right)}}$with respect to the parameters (x_(i), y_(i)), i=1, . . . , N of eachlayer v that composes the CNN. The objective function is differentiablew.r.t. the parameters k_(v), v=1 . . . ∨V ∨ and the stochastic gradientdescent method incrementally updates the parameters k_(v), v=1 . . . ∨V∨ by feeding batches of images.

Each CNN may be trained jointly in a computationally efficient manner interms of speed and memory consumption as follows. First we let themshare the same convolutional layers. Only the last layers differ so thateach CNN learn a specific global scene descriptor. Second, we define amulti-objective loss function as a (possibly weighted) sum of all theerrors

${\ell\left( {\begin{bmatrix}{{CNN}_{1}(x)} \\\vdots \\{{CNN}_{K}(x)}\end{bmatrix},\begin{bmatrix}c_{1} \\\vdots \\c_{K}\end{bmatrix}} \right)} = {\sum\limits_{k = 1}^{K}\;{w_{k}{\ell_{k}\left( {{{CNN}_{k}(x)},c_{k}} \right)}}}$

Each CNN_(k) corresponds to the locale predictor, mood predictor and soon. They are applied to the image tensor x to estimate either a globalscene probability vector or a regional probability tensor CNN_(k) (x).Each loss function l_(k) evaluates the distance between the estimatetensor CNN_(k)(x) and the ground-truth tensor c_(k). Thus during thetraining stage, the back propagated errors from the multi-objective lossfunction enables the weights of the shared convolutional layers tobecome optimal w.r.t. all the classification tasks.

Like the regional context probability tensor, we define the insertionprobability tensor as follows.

Definition 5 An insertion probability tensor a is defined as a tensor ofprobability vectors in [0,1]^(N×C′×H×W) where a[n,k,i,j] quantifies aconfidence value for a class of insertion descriptor.

The insertion probability vector can just encode the insertion objectembed type for example, vehicle, soda bottle, cell phone, etc. or notsuitable for object insertion. Each entry a_(n) [., i, j] encodes theconfidence value that, for example, pixel x_(n)[.,i,j] is:

k=1: not suitable for object insertion advertising,

k=2: suitable for insertion of a vehicle type of object productplacement,

k=3: suitable for insertion of a soda bottle type of object signageplacement,

k=4: suitable for insertion of a cell phone type of object.

And so on.

It will be appreciated that this is just one particular example of thetypes of object that may be identified in the insertion probabilityvector and that any number of additional or alternative types of objectmay be identified in the insertion probability vector.

The above definitions have helped to explain how the corpus of trainingimages may be annotated, and consequently how a trained machine learningsystem may then annotate the plurality of frames of the source video(for example, the scene descriptor sub-module 610 may be trained toannotate a global context probability vector and/or a regional contextprobability vector(s) for each pixel of the frames in the way describedabove in relation to the scene descriptor probability vector, and theidentification sub-module 620 may be trained to annotate each pixel ofthe frames with an insertion probability vector described above).Therefore, we shall now briefly describe ways in which the machinelearning training may be carried out.

Interpreting Feature Maps in the Recognition of Global Scene Descriptors

We show below VGG-16, an example of CNN architecture used for imageclassification for 1000 classes.

FIG. 11 shows the intermediate results of the CNN at different stagesupon feeding an image. In this particular classification task, the inputof the CNN is an image and is represented as a 3D volume of width 224,of height 224, of depth 3.

The output is the softmax block which is 1000D probability vector.

The computational flow in a CNN is as follows:

-   -   The image is first transformed into a feature map 224×224×64        after the first convolution+ReLU block. The feature map        describes each pixel (i,j) ∈ [1, 224]×[1, 224] of the image with        a 64D feature vector as the result of 64 different convolutional        kernels.    -   The first feature map 224×224×64 is transformed into a second        feature map 224×224×64 after a second convolution+ReLU block.        Again the second feature map describes each pixel (i,j) ∈ [1,        224]×[1, 224] of the image with a 64D feature vector as the        result of 64 different convolutional kernels.    -   The second feature map is then transformed by a max-pooling        layer into a third feature map 112×112×64. The feature map can        be interpreted as a grid of 112×112 image blocks. Each block        (i, j) corresponds to a non overlapping image patch (i, j) of        2×2 pixels in the original image. Each block is described by a        64D feature vector (not 128D as the image would mislead to.)    -   The third feature map is then transformed by a convolution+ReLU        block into a fourth feature map 112×112×64. Each block (i, j)        corresponds to a non overlapping image patch (i,j) of 2×2 pixels        in the original image and is described by a 128D feature vector        as the result of 128 convolution kernels.    -   And so on as the reader will appreciate how the remaining        feature maps are generated by pursuing the reasoning above.

Consequently we easily understand that a CNN builds a multiscalerepresentation due to the max-pooling operation. In the case of VGG-16,we observe namely at the end of each max-pooling function, the image issuccessively represented as:

-   -   a grid of 112×112 image blocks, each block describing a        non-overlapping image patch of 2×2 pixels of the original image        with a 64D vector;    -   a grid of 56×56 image blocks, each block describing a        non-overlapping image patch of 4×4 pixels with a 256D feature        vector;    -   a grid of 28×28 image blocks, each block describing a        non-overlapping image patch of 8×8 pixels with a 512D feature        vector;    -   a grid of 14×14 image blocks, each block describing a        non-overlapping image patch of 16×16 pixels with 512D feature        vector.

After which, the coarsest grid of 14×14 image blocks is used eventuallytransformed into a 1000D probability vector from the last layers beingcomposed of inner-product, dropout and softmax layer altogether formingwhat is called a perceptron network.

Recognition of Regional Context Descriptors

To compute a specific regional context probability vector, the originalVGG-16 architecture is not directly suited to perform pixelclassification. However we have pointed out previously that VGG-16builds a multiscale (or pyramidal) representation of the input image. Asa first approach, every pixel of the original image can be described byconcatenating the feature vector at every layer of the pyramid.

Intuitively, the sole colour value of the pixel not always enoughwhether it corresponds to a skin pixel, because the skin colour is notuniform. However if we analyse the mean colour of the neighbouringpixels with varying neighbourhood size, it becomes more and more obviousto the CNN model to infer that the pixel is indeed a skin pixel.

Fully convolutional networks and variant networks exploit and refinethis intuition with deconvolutional layers.

Human Action Recognition via LSTM network

It is convenient to describe the human activity by means of sentence anda LSTM is designed to predict a sequence of words. To enable the machineto predict such a sequence of words, it suffices to replace theperceptron network by a LSTM network. Unlike usual layers, the LSTMmaintains a state, encoded by a cell state vector. This vector can bethought as a ‘memory’ continuously built from the past predictions andthis is one aspect of the LSTM that ensures the temporal coherence ofpredictions.

The LSTM is updated by a set of transition matrices and weight matrices.The matrices are the parameters optimized during the training stage. Onerole of these matrices is to update the cell state vector (the ‘memory’)by appropriately weighting the importance of the new prediction. We willnot detail further the mathematical mechanisms of the LSTM network andthe reader should just understand that an LSTM is just anotherdifferentiable function. Thus a vanilla stochastic gradient methodduring the training stage works as usual.

Experimentally such a network using VGG-16+LSTM has shown impressiveresults in the automatic captioning of images using.

Recognition of Insertion Descriptors

To recognise insertion descriptors, we employ a SVM-based approach. ASVM is a classification algorithm useful for predicting whether anobject belongs to a particular class, and may be used in supervisedlearning applications. A SVM-based classifier can only perform a binaryclassification. While it may seem a limitation, it can be generalised toa robust multiclass classification as follows.

In the indirect approach, we train a dedicated SVM classifier for eachclass of brand category, for example, “Kitchen Appliances”, using aone-vs-all strategy, where the training data is composed of positivesamples, i.e., images relevant for “kitchen appliances”, and negatives,i.e., images irrelevant for “kitchen appliances”.

After the training stage, each class-specific classifier computes aprediction score for a new unseen image. It should provide a positivescore when the image is suitable for such a brand category and anegative score when it is not. The higher the score the more suitablethe image is for the brand category. It is then possible to establish aranking of brand categories. One advantage of using SVM instead of CNNis that we can incrementally learn to recognise a new brand categorywithout having to start the learning process from scratch. Anotheradvantage is that SVM will behave better than CNN where the classes arenot mutually exclusive. For the brand category classification problem, ascene can be suitable for many brand categories indeed. However unlikeCNN, the SVM is unable to learn a transformation of the image data intoan efficient feature vector. Rather a SVM requires a featurerepresentation beforehand to ensure good prediction results for theappropriate recognition task.

Semi-Supervised Learning for Less Labour-Intensive Annotation

There are some ways to train learning systems. The easiest but mostlabour-intensive approach is the supervised learning approach where eachtraining sample are required to be fully annotated. In particular, forthe prediction of regional context descriptor, every pixel of the imagemay be annotated. The hardest but less labour-intensive approach is thesemi-supervised learning approach.

Obtaining annotations for each training video shot is an expensive andtime-consuming task. In practice, it may be more efficient not toannotate every single pixel for regional context vector and insteadprovide a not necessarily complete yet sufficient amount of annotations.

In particular we may want to allow the training to contain loosely orpartially annotated video shots, e.g., bounding boxes, scribbles.Semi-supervised learning algorithms tackles such problems.

Temporal Coherence using LSTM

Video Networks. The submodule 610 may be extended to Video data ratherthan on image frames because of the generality of convolutional neuralnetworks. However, video networks are not practical. Most importantly itraises the question of appropriately video data along the temporaldimension which potentially means losing information and a drop ofaccuracy in the prediction task.

LSTM and Variants. Instead it is in practice more efficient to use LSTMnetwork to ensure temporal coherence instead of a perceptron network.The LSTM remains applicable to locale detection, mood detection,regional context descriptor, blue box prediction as it simply means toreplace the perceptron network by a LSTM network in each correspondingCNN. Notice that they are numerous variant methods that borrows the sameprinciple of LSTM in the semantic segmentation tasks. Let us mention forexample the clockwork approaches.

FIG. 8 shows example steps of a process 800 for the training the machinelearning of the scene descriptor sub-module 610 and identificationsub-module 620 in order to determine scene descriptors as describedearlier. The scene descriptor sub-module 610 may comprise a CNN, whichin step S802 is provided with a corpus of training images that areannotated with scene descriptor probability vectors as described above.This is to provide the CNN with the means to learn the features ofimages which can be associated with scene descriptors, as describedabove. The last layers of the CNN may be used to extract generic visualrecognition features relating to regional context descriptors and/orglobal context descriptors. The neural network model comprising weightsand predictions for the scene descriptors is developed in step S804. Theidentification sub-module may comprise an SVM model for identifyingcandidate insertion zones and a further CNN/CVM model for determiningcorresponding candidate object types. The neural network modelcomprising weights and predictions may be provided to the identificationsub-module 620 to train an SVM to predict the most useful scenedescriptors for use in determining the candidate object types in stepS808. Prior to providing the generic visual recognition features createdfrom the activations in the last layers of the CNN to the SVM, variouspre-processing steps S806 may be implemented to refine the process atthe SVM stage. This can include L2 normalising the features, orcombining the features from different patches in the image.

Having trained the scene descriptor sub-module 610 and identificationsub-module 620, they may then process the image contents of theplurality of frames of the source video as follows:

-   -   1. a CNN-based model in the scene descriptor sub-module 610        generates for the scene a heat map for each scene descriptor        (for example, by determining a regional context probability        vector for each pixel the plurality of frames, in which case a        2D heat map of regional context probability would be generated        for each frame, with the temporal element by virtue of the heats        maps for the plurality of frames);    -   2. an SVM model in the identification sub-module 620 which then        identify the candidate insertion zones within the image contents        based on the scene descriptors;    -   3. a further CNN/SVM model in the identification sub-module 620        then determines the corresponding insertion descriptors for each        candidate insertion zone.

Direct Approach

As explained above in respect of the “indirect approach”, there may be acorrelation between particular scene descriptors and types of objectthat are suitable for insertion into the scene. However, it has beenrealised that in some instances, different scene descriptors may beorthogonal for two reasons:

-   -   For example, let us consider placing a wine bottle on a dining        table. From a purely context point of view, the association        table-bottle would appear more correct than the association        wall-bottle. Therefore, every table pixel may be considered more        relevant than a wall pixel for a wine bottle placement.        Consequently, a correlation between table and bottle may be        inferred, whereas a correlation between wall and bottle may not.

From the point of view of a content analyst or an embed artist, however,it may be slightly subtler than that. First, because of the 3D geometry,the placed bottle will need to occupy at least some table pixels andpossibly some wall pixels. Second, not every single table pixel has theobject insertion impact: if a character is sitting at a dining table, itmay have more impact inserting the bottle next to the character's hand,rather than at the other end of the table.

-   -   Learnable Statistical Properties of Insertion Zones. Our data        show that insertion zones suitable for object insertion        identified by content analysts are very often dependent on their        positioning relative to other ‘things’ in the image contents.        For example, they may choose insertion zones that are “parts of        a table that are close to arms and hands of characters”.        Likewise, signage placement opportunities may very often of the        type, “outdoor building walls” rather than “indoor walls”.

Furthermore specific object types relevant to different types ofsurface, for example table-top, work surface, and bar counter, can belearnt jointly.

These two observations have a non-trivial consequence. Whilst the scenedescriptors described above in relation to the “indirect approach” maybe very useful, they may not actually be necessary for identifyingcandidate insertion zones and determining candidate object types thatare suitable for insertion into the candidate insertion zone. A machinelearning system, for example one using Deep Neural Networks, may be ableto capture the striking statistical properties of insertion zones and,therefore, simultaneously identify candidate insertion zones anddetermine candidate object types for those identified candidateinsertion zones. This is referred to in the present disclosure as the“direct” approach, since machine learning is used to identify anddetermine the candidate insertion zones and candidate object typesdirectly, in a single step, from processing the image content of theplurality of frames (in contrast to the “indirect” approach, where theimage contents of the plurality of frames are first processed usingmachine learning to determine the scene descriptors, and the candidateinsertion zones and candidate object types then determined in a secondmachine learning step from the scene descriptors).

FIG. 9 shows an example schematic representation of a configuration ofthe candidate insertion zone module 110 for carrying out the “directapproach”. As can be seen, the insertion zone and insertion objectidentification sub-module 910 receives the plurality of frames of ascene and processes the image contents of the frames to identify thecandidate insertion zone(s) and the one or more candidate object types.

The insertion zone and insertion object identification sub-module 910may comprise a CNN model that may be trained in a similar way to thatdescribed above. In this way, the insertion zone and insertion objectidentification sub-module 910 may be able to learn what sort of imagecharacteristics (for example, types of scene descriptors, relativepositioning of regional context descriptors) may determine the size andpositioning of insertion zones, and in turn may lend themselves to theinsertion of particular types of object. Since in the training corpusobjects will typically have been inserted into the image contents forparticular reasons, for example particular object types will have beeninserted into the image because they fit in well with the rest of theimage contents and/or objects may be inserted closer to particularcharacters in order to increase the impact of the inserted object (asexplained earlier), the insertion zone and insertion objectidentification sub-module 910 should inherently learn this from thetraining corpus. Consequently, when the trained insertion zone andinsertion object identification sub-module 910 processes the pluralityof frames of a new source video, it may naturally identify candidateinsertion zones to be in the best regions of the image contents (forexample, in the table and wall pixels close to a character's hand forthe insertion of a wine bottle, rather than in table pixels well awayfrom a character's hand, as described earlier in the ‘indirect’ approachsection).

Similarly to the identification sub-module 620 described earlier, theinsertion zone and insertion object identification sub-module 910 mayoutput an annotated version of the plurality of frames, the annotationscomprising an insertion probability vector for each pixel. The postprocessing sub-module 920 may be configured to operate in the same wayas the post-processing sub-module 640 described earlier and output anidentification of the candidate insertion zone and correspondinginsertion descriptor as described earlier. However, the post-processingsub-module 920 is optional and in an alternative, the candidateinsertion zone module 110 may simply output the annotated plurality offrame generated by the insertion zone and insertion objectidentification sub-module.

In the above described “direct” and “indirect” implementations, thetraining of the machine learning modules is carried out using a corpusof training images that are annotated with scene descriptors andinsertion descriptors. However, in some instances, a sufficiently largebody of training material comprising these annotations may not beavailable. For example, there may be a large corpus of images that havebeen annotated by a content analyst or embed artist with insertiondescriptors, but not any scene descriptors, since the content analyst orembed artist may have been tasked only with inserted objects into thoseimages. In this case, the “direct” approach may still be effective,since it may still implicitly learn the different characteristics of theimages that have led to the content analyst or embed artist to choosethe insertion zone and insertion object that they have chosen. However,it may still be preferable for the machine learning module to learn howto recognise scene descriptors for images in order further to improveits identification of candidate insertion zones and determination ofcandidate object types. In this case, where a training corpus comprisingonly insertion descriptors is available, other trained machine learningmodules may be utilised as part of the training process.

FIG. 10 shows an example representation of a training system comprisinga trained machine learning module 1010 and a machine learning module tobe trained 1020. The machine learning module to be trained 1020 may bethe scene descriptor sub-module 610 and identification sub-module 620 ofthe “indirect approach” above, or the insertion zone and insertionobject identification sub-module 910 of the “direct approach” above. Inthis example, a training corpus annotated with insertion zonedescriptors is available. This is fed to both the trained machinelearning module 1010 and the machine learning module to be trained 1020.The trained machine learning module 1010 may be trained to identifyscene descriptors (for example, it may be trained to perform regionalcontext recognition), such that it can identify scene descriptors forthe training corpus of images and feed those to the machine learningmodule to be trained 1020 (for example, as with scene descriptorprobability vector annotations of the training corpus of images). Thus,the machine learning module to be trained 1020 may still be trained tooperate as described earlier using a training corpus of images thatlacks scene descriptors, by utilising an existing trained machinelearning module 1010.

Optionally, for both the direct and indirect approaches described above,an operator or user may provide feedback on the identified candidateinsertion zone and/or insertion zone descriptor to the candidateinsertion zone module 110. This optional implementation is representedin FIG. 12, which is very similar to FIG. 1, but includes the additionaluser/operator feedback.

A user or operator may review the identified candidate insertion zoneand/or insertion zone descriptor in any suitable form (for example, byreviewing the object insertion suggestion frame and/or the insertionzone suggestion frame, etc) and assess its suitability for the imagecontents of the plurality of frames. In this way, a skilled operator oruser may utilise their object insertion expertise to appraise thesuitability of the candidate insertion zone and/or insertion zonedescriptor that has been determined at least in part using machinelearning.

The feedback may take any suitable form, for example the user mayindicate if the identified candidate insertion zone and/or insertionzone descriptor are suitable or unsuitable for the image contents of theplurality of frames, or they may rate the suitably, for example on ascale of 0-5 or 0-10 or 0-100, etc. The feedback may then be used toimprove the machine learning algorithms that have been used in thecandidate insertion zone module 110, so that the quality or suitabilityof the candidate insertion zone and/or insertion zone descriptordetermined in the future may be improved.

The skilled person will readily appreciate that various alterations ormodifications may be made to the above described aspects of thedisclosure without departing from the scope of the disclosure.

For example, optionally, the system 100 may further comprise a finalinsertion module configured to receive an object or additional materialfor insertion into the scene of the source video and generate outputmaterial comprising at least part of the source video and the receivedobject or additional material inserted into the candidate insertionzone. The received object or additional material may be of the typeindicated by the candidate object type. The object or additionalmaterial may be received, for example, from a data store/library ofadditional material (which may be part of, or separate from, the system100) by virtue of retrieval based on the insertion zone descriptor, orby any other means. In this way, the final insertion module may functionsimilarly to the object insertion module 140, as described above, butrather than create an object insertion suggestion frame, it may actuallyinsert the object into the image content of the plurality of frames ofthe scene. The insertion itself may take place according to any standardtechniques that would be well understood by the skilled person. Thereceipt and insertion of the object or material may be automatic, or maytake place after receiving approval from a user who has considered thecandidate insertion zone and the type of object that recommended asbeing suitable for insertion into the candidate insertion zone. In thisway, a suitable object or additional material may be inserted into theimage contents of a scene quickly and reliably.

Where the insertion is automatic, the system 100 may be configured suchthat it's only output is the output material comprising the object oradditional material inserted into the candidate insertion zone. Wherethe insertion takes place after user approval, the system 100 may outputat least one of: an identification of the candidate insertion zone andcandidate object types; the objective insertion suggestion frame; and/orthe insertion zone suggestion frame. After receipt of user approval, thesystem 100 may then output the output material comprising the object oradditional material inserted into the candidate insertion zone.

Furthermore, FIGS. 1, 6, 9 and 10 comprise various interconnectedmodules/entities. However, the functionality of any two or more of themodules/entities may be performed by a single module, for example thefunctionality of the candidate insertion zone module 110 and the objectinsertion module 140 may be implemented by a single entity or module.Likewise, any one or more of the modules/entities represented theFigures may be implemented by two or more interconnected modules orentities. For example, the functionality of the scene descriptorsub-module 610 may be implemented as a system of interconnected entitiesthat are configured to together perform the functionality of the scenedescriptor sub-module 610. The entities/modules represented in theFigures (and/or any two or more modules that may together perform thefunctionality of an entity/module in the Figures) may be co-located inthe same geographical location (for example, within the same hardwaredevice), or may be located in different geographical locations (forexample in different countries). They may be implemented as only a partof a larger entity (for example, a software module within amulti-purpose server or computer) or as a dedicated entity.

The aspects of the disclosure described above may be implemented bysoftware, hardware, or a combination of software and hardware. Forexample, the functionality of the candidate insertion zone module 110may be implemented by software comprising computer readable code, whichwhen executed on the processor of any electronic device, performs thefunctionality described above. The software may be stored on anysuitable computer readable medium, for example a non-transitorycomputer-readable medium, such as read-only memory, random accessmemory, CD-ROMs, DVDs, Blue-rays, magnetic tape, hard disk drives, solidstate drives and optical drives. The computer-readable medium may bedistributed over network-coupled computer systems so that the computerreadable instructions are stored and executed in a distributed way.Alternatively, the functionality of the candidate insertion zone module110 may be implemented by an electronic device that is configured toperform that functionality, for example by virtue of programmable logic,such as an FPGA.

FIG. 13 shows an example representation of an electronic device 1300comprising a computer readable medium 1310, for example memory,comprising a computer program configured to perform the processesdescribed above. The electronic device 1300 also comprises a processor1320 for executing the computer readable code of the computer program.It will be appreciated that the electronic device 1300 may optionallycomprise any other suitable components/modules/units, such as one ormore I/O terminals, one or more display devices, one or more furthercomputer readable media, one or more further processors, etc.

It is claimed:
 1. A system, including a processor and memory,comprising: the processor executing instructions which cause theprocessor to operate as a candidate insertion zone module, the candidateinsertion zone module configured to: receive a plurality of frames of ascene of a source video, wherein each of the plurality of framescomprises a plurality of pixels; and process, at least in part usingmachine learning, image contents of the plurality of frames to:determine, for at least some of the pixels of the plurality of frames ofthe scene, an insertion probability vector comprising a probabilityvalue for each of a plurality of insertion labels, each probabilityvalue being indicative of the likelihood that the type of insertionindicated by the corresponding insertion label is applicable to thepixel, wherein the insertion probability vectors are indicative of acandidate insertion zone for the insertion of an object into the imagecontent of at least some of the plurality of frames and a candidateobject type indicative of a type of object that is suitable forinsertion into the candidate insertion zone.
 2. The system of claim 1,wherein the instructions further cause the candidate insertion zonemodule to operate including an identification sub-module, theidentification sub-module configured to: perform the determination ofthe insertion probability vectors.
 3. The system of claim 2, wherein theplurality of insertion labels comprises: a label indicative of the pixelnot being suitable for insertion of an object; and one or more labelsindicative of a corresponding one or more types of object.
 4. The systemof claim 2, wherein the candidate insertion zone comprises a pluralityof pixels having insertion probability vectors that all have a maximumargument of probability values corresponding to a label that isindicative of the candidate object type.
 5. The system of claim 2,wherein the instructions further cause the candidate insertion zonemodule to operate including a scene descriptor sub-module, the scenedescriptor sub-module configured to: process, using machine learning,image contents of at least some of the plurality of frames to determinea scene descriptor, wherein determination of the insertion probabilityvectors is based at least in part on the scene descriptor.
 6. The systemof claim 5, wherein: determination of the insertion probability vectorsis based at least in part on the scene descriptor.
 7. The system ofeither claim 5, wherein the scene descriptor comprises at least oneregional context descriptor indicative of an identified entity in thescene.
 8. The system of claim 7, wherein the scene descriptor sub-moduleis configured to process, using machine learning, image contents of theplurality of frames to determine, for at least some of the pixels of theplurality of frames of the scene, a regional context probability vectorcomprising a probability value for each of a plurality of regionalcontext labels, each probability value being indicative of thelikelihood that the type of entity indicated by the correspondingregional context label is applicable to the pixel.
 9. The system ofclaim 8, wherein the plurality of regional context labels comprises: alabel indicative of the pixel not relating to anything; and at least oneof: one or more labels indicative of a human; one or more labelsindicative of an animal; one or more labels indicative of an object; oneor more labels indicative of a surface.
 10. The system of claim 1,wherein the instructions further cause the candidate insertion zonemodule to operate including an insertion zone and insertion objectidentification sub-module, the insertion zone and insertion objectidentification sub-module configured to: perform the determination ofthe insertion probability vectors.
 11. The system of claim 10, whereinthe plurality of insertion labels comprises: a label indicative of thepixel not being suitable for insertion of an object; and one or morelabels indicative of a corresponding one or more types of object beingsuitable for insertion in the pixel.
 12. The system of claim 10, whereinthe candidate insertion zone comprises a plurality of pixels havinginsertion probability vectors that all have a maximum argument ofprobability values corresponding to a label that is indicative of thecandidate object type.
 13. The system of claim 1, wherein theinstructions cause the processor to operate as a segmentation module,the segmentation module configured to: generate an insertion zonesuggestion frame comprising a frame of the plurality of frames overlaidwith a visualisation of the candidate insertion zone.
 14. The system ofclaim 1, wherein the instructions cause the processor to operate as anobject insertion module, the object insertion module configured to:select an object for insertion based on the candidate object type; andgenerate an object insertion suggestion frame comprising a frame of theplurality of frames and the selected object inserted in the candidateinsertion zone.
 15. The system of claim 1, wherein the instructionsfurther configure the candidate insertion zone module to: receivefeedback from an operator, wherein the feedback is indicative of thesuitability of the identified candidate insertion zone and/or thecandidate object type for the image contents of the plurality of frames;and modify the machine learning based at least in part on the feedback.16. A method of processing the image contents of a plurality of framesof a scene of a source video, the method comprising: receiving theplurality of frames of the scene of the source video; and processing, atleast in part using machine learning, image contents of the plurality offrames to: determine, for at least some of the pixels of the pluralityof frames of the scene, an insertion probability vector comprising aprobability value for each of a plurality of insertion labels, eachprobability value being indicative of the likelihood that the type ofinsertion indicated by the corresponding insertion label is applicableto the pixel, wherein the insertion probability vectors are indicativeof a candidate insertion zone for the insertion of an object into theimage content of at least some of the plurality of frames and acandidate object type indicative of a type of object that is suitablefor insertion into the candidate insertion zone.
 17. A computer programfor carrying out the method of claim 16 when executed on the processorof an electronic device.
 18. A method of training a candidate insertionzone module to identify candidate insertion zones and one or morecandidate objects for insertion in a scene of a source video, the methodcomprising: receiving a training corpus comprising a plurality ofimages, each annotated with identification of at least one insertionzone and one or more candidate object types for each insertion zone; andtraining the candidate insertion zone module using machine learning andthe training corpus to process image contents of a plurality of framesof the source video to: determine, for at least some of the pixels ofthe plurality of frames of the scene of the source video, an insertionprobability vector comprising a probability value for each of aplurality of insertion labels, each probability value being indicativeof the likelihood that the type of insertion indicated by thecorresponding insertion label is applicable to the pixel, wherein theinsertion probability vectors are indicative of a candidate insertionzone for the insertion of an object into the image content of at leastsome of the plurality of frames and a candidate object type indicativeof a type of object that is suitable for insertion into the candidateinsertion zone.
 19. The method of claim 18, wherein at least some of theplurality of images in the training corpus are each further annotatedwith a scene descriptor, and wherein the candidate insertion zone moduleis further trained using machine learning to: identify at least onescene descriptor for the image content of at least some of the pluralityof frames; and determine the insertion probability vectors based atleast in part on the identified at least one scene descriptor.
 20. Themethod of claim 18, further comprising: determining one or more scenedescriptors for at least some of plurality of images in the trainingcorpus using a trained machine learning module configured to identify ascene descriptor by processing the content of an image; wherein trainingthe candidate insertion zone module using machine learning furthercomprises training the candidate insertion zone module to: identify atleast one scene descriptor for the image content of at least some of theplurality of frames; and determine the insertion probability vectorsbased at least in part on the identified at least one scene descriptor.